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Abstract 

Assessment drives significant change in higher education. 
Colleges and universities are being asked to (a) publish expected 
learning outcomes for each of their programs; (b) provide 
evidence that the expected learning outcomes are realized by 
students; and (c) demonstrate how such data collection and 
analyses lead to continuous improvement of student learning, 
the curriculum, and the university. This is a significant challenge, 
especially for large schools with many students. 

This paper presents (a) a working yet evolving model of an 
institutional assessment and analysis plan; (b) obstacles and 
challenges associated with implementing the plan; (c) innovative 
strategies — some technological — employed to address the 
obstacles and challenges; (d) a collegewide application of a 
systems approach to assessment; and (e) lessons learned and 
future strategies. Institutional, collegewide, discipline-specific, 
and individual faculty perspectives to a systems approach of 
bridging assessment, learning outcomes, and accreditation will be 
presented. The importance of student involvement throughout 
the process is highlighted. 

Brigham Young University (BYU) serves over 30,000 students 
across 1 1 colleges and 56 academic departments, offering over 
400 degree-granting programs. During BYU's recent decennial 
regional accreditation, emphasis on assessment strategies and 
learning outcomes accountability was appropriately far reaching 
and has generated extensive response across the university 
community. 

Introduction 

On the surface, few would argue that the reason we have 
institutions of higher education is to facilitate learning. This 
sounds simple enough; however, pressure is surfacing from 
multiple directions, for instance, national educational entities. 
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regional accrediting bodies, discipline-specific 
specialized accrediting entities, funding agencies, 
and key stakeholders (students, employers, parents, 
institutional boards, etc.) for evidence of increased 
levels of learning to validate the very high costs 
of academia, most of which are on the rise. Due 
to these pressures, both internal and external to 
the institution, many institutions find themselves 
asking how to provide evidence of learning in a 
consistent, timely manner to satisfy these demands 
and to inform and influence a systematic and 
continuous process of improvement and change 
at the institution. The authors acknowledge that 
assessments that contribute to decision-making 
in all key performance areas of the institution 
are extensive and involve a variety of challenges 
and constraints. Assessment becomes especially 
challenging when consideration must be given 
to the place of assessments and reporting in the 
larger context of aligning institutional claims with 
outcomes projected for all performance areas of the 
campus (Kramer, Hanson, & Olsen, 201 0). 

Given the difficulties and challenges of 
assessment at the institutional level, an institutional 
assessment plan is absolutely necessary to 
accomplish the task of providing evidence that 
demonstrates learning and that also ensures 
decisions informed by these data positively 
promote and improve student learning and 
development — the core purpose of assessments 
in higher education. In addition, individual 
academic units and educational support units 
at the university must also develop their own 
assessment plans, however, the institutional plan 
may provide guidance for all entities. Such plans 
are integral to an institution's ability to inculcate a 
culture of assessment wherein unit- or program- 
level goals and objectives can be rolled up to the 
institutional level, thus assisting the validation 
of institutional core values and corresponding 
objectives. Assessment needs to be by design, not 
happenstance, or random coincidence. Simply put, 
he who fails to plan, plans to fail. Every unit on a 
campus, whether academic or educational support, 
has a genuine role in substantiating the degree to 
which the institution's mission is being achieved. A 



well-designed institutional plan proactively makes 
such important connections and efforts. 

This paper profiles Brigham Young University's 
(BYU) recent efforts in the quest for a deepened 
culture of assessment in the pursuit of increased 
learning and ongoing improvement by addressing 
the following: 

• BYU's Institutional Assessment and Analysis 
Plan; 

• Challenges and Strategies; 

• Collegewide Application of a Systems 
Approach to Assessment; and 

• Lessons Learned and Future Strategies. 

While no panacea is offered for creating and 
managing an institutionwide assessment plan 
tailored to a university or college, the article does 
offer essential, established principles based on 
working models and assessment frameworks that 
have succeeded at BYU. In sum, this paper offers 
ideas on how to move towards a stronger, more 
congruent institutional assessment and analysis 
plan. 

BYU'S Institutional Assessment and 
Analysis Plan 

BYU serves over 30,000 students across 1 1 
colleges and 56 academic departments, offering 
over 400 degree-granting programs. The work of 
establishing a universitywide assessment plan for 
such a large institution cannot be understated, but 
an institution-level assessment plan is not complete 
without ongoing assessment in its individual 
academic and educational support programs. 
Institution-level assessment is complementary 
to lower level (program, course, etc.) assessment 
and vice versa. As a result, many of the strategies 
discussed below were also undertaken to 
aid individual programs and departments in 
establishing their own assessment program. 

Despite the wealth of information regarding 
BYU's specific strategic plan for assessment, the 
authors realize that strategic plans are as individual 
as the institutions that develop them. So, we do not 
make the assumption that all of the suggestions 
and strategies discussed will be applicable to all 
institutions, but should be taken in the context 
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of one's own institution or program. The plan 
presented is not intended to be the "model" 
for all institutions, but to generate thoughtful 
consideration of how other institutions may tackle 
comparable issues. 

During BYU's recent decennial regional 
accreditation, emphasis on assessment strategies 
and learning outcomes accountability was 
appropriately far-reaching and generated extensive 
response across the university community. In 
response to this experience, the university set out 
to enhance aspects of the current institutional 
assessment plan. Administrators and faculty at BYU 
have worked tirelessly to establish a strategic plan 
for assessment that works for the institution and 
also complements its programs' assessment plans. 
While there is too much information about the 
specifics of BYU's strategic plan to include it all in 
this paper, the key principles of BYU's strategic plan 
for assessment include the following aspects: 

Align the assessment plan to institutional 
mission, goals, and learning outcomes. As 
institutions undertake the process of developing 
an assessment plan, they may find themselves 
asking what really matters in assessment or what 
are purposeful, intentional assessments. Moreover, 
in order to achieve meaning in assessment, 
instruments and measures must be aligned to the 
claims and goals the institution makes. How can 
institutions make assessments more meaningful? 
The answer according to Suskie (2009) and Banta, 
Jones, and Black (2009), is found in placing value in 
claims institutions make, in the goals and missions 
they state, and in the learner outcomes they project. 

Institutional goals, mission statements, and 
expected learning outcomes are the foundation 
upon which all targeted learning at the course 
and program levels must be based. Outcomes or 
goals at the program level combine with university 
core education goals. In other words, goals at the 
institution, program, and course (or activity) ought to 
be interconnected, complementary, and reciprocal. 

The alignment of learning outcomes with 
program and institutional goals at BYU is 
pursued from a number of efforts integrated 
with the institutional assessment plan. First, the 



alignment between the learning outcomes to 
stated goals is mapped and documented via the 
learningoutcomes.byu.edu web portal. This portal 
provides a common template wherein learning 
outcomes are recorded and authenticated by 
faculty. Recent enhancements to this system have 
included (a) incorporating the mapping of course- 
level learning objectives and (b) a course-level 
matrix mapping of courses, which are expected 
to contribute to a student's development in the 
prescribed program level and expected learning 
outcomes. Second, the BYU Office of Institutional 
Assessment and Analysis has developed a suite of 
BYU instruments capturing transition-point data 
from students as they move through their BYU 
experience. Each of these instruments is aligned 
with 24 constructs, which operationalize the stated 
mission and aims of the university. 

Commit to and invest in assessment. Kramer 
et al. (2010) observed that "the research literature 
emphasizes the need for improvement in leading 
assessments by means of a strategic assessment 
plan to measure learner outcomes, claims, and 
goals and to improve student learning and 
programs from what is measured" (p. 44), which 
is supported by other researchers (Angelo, 2007; 
Banta & Associates, 2002; Banta et al., 2009; 

Bresciani, Gardner, & Hickmott, 2009; Bresciani, 
Zelna, & Anderson, 2004; Ewell, 2005, 2007; Maki, 
2004; Schuh & Associates, 2009; Suskie, 2007, 2009; 
Walvoord, 2004; Wehlburg, 2008). However, Kramer 
et al. (201 0) further state that this sounds much 
more simple than it actually is, as institutions are 
aware. Implementing and sustaining a strategic 
assessment plan requires commitment and a 
willingness to fully invest in the plan from all entities 
involved (see p. 29). Associated investments include 
expenditures of financial, human, organizational, 
political, and symbolic capital (e.g., elements 
represented in dollars and time). Despite the high 
costs of supporting an assessment plan or system 
at the institution, assessment is not a "money pit" 
sort of business. "Effective assessment can support 
fiscal management at a university, resulting in an 
increased return on precious campus resources" (see 
Kramer et al., 201 0, p. 46). 
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Build a shared responsibility for the 
assessment plan through engaging stakeholders. 

Everyone on campus should in some way or another 
share accountability in an institution's assessment 
strategy focused toward improvement. Academic 
units as well as educational support units (e.g., 
bookstore, library, housing, employment, student 
life) contribute to students' university experiences, 
their subsequent learning, and their perceived 
return on their educational investment. 

Many faculty are well invested in the notions 
of learning outcomes and related assessments; 
however, some feel that assessment, especially 
the kind that satisfies external accrediting bodies 
or other stakeholders external to the classroom, 
is someone else's job. It is a significant challenge 
to genuinely convince faculty that their return on 
investment in classroom assessment is worth any 
extra perceived effort beyond what they are already 
doing. Our experience at BYU with a good number 
of faculty (though not all) has been that successful 
motivation for meaningful participation in such 
endeavors has been best accomplished by focusing 
the primary purpose around students and their 
learning. Generally, BYU faculty members have had 
a more positive demeanor towards assessment 
if the purpose is framed around their students 
and their learning than if it is framed around 
accreditation or other external accountabilities. 
Thus, one of our strategies was to focus on students 
and their learning. Faculty feel overworked and 
stressed, lacking the capacity for anything else to 
be added to their loads. Some honestly wonder 
what all the fuss is about and why giving grades 
isn't sufficient. Others wonder what difference it 
really makes anyway. The art is convincing faculty 
who struggle with learning assessment that well- 
structured and targeted assessment embedded 
in their pedagogy, once established seamlessly 
integrates everything into the status quo — and is 
not a painful add-on. 

Use the assessment results to effect change. A 

commitment to the assessment of student learning 
requires a parallel commitment to ensuring its 
use. Perhaps the most difficult part of assessing 
student learning is the process of effecting change 



in teaching and learning as a result of information 
gained through assessment practices. It is pointless 
to only "do assessment"; the results of assessment 
activities should come full circle to have a direct 
impact on teaching and learning and on the 
institution's strategic plan to fulfill its mission 
(Middle States Commission on Higher Education, 
2003 ). 

Changes in programmatic curricula as a result 
of assessment data do not happen automatically, 
as many can attest. However, if plans at the 
department level outline specific procedures for 
examining assessment results and implementing 
curricular revision, those changes are more 
likely to occur. Assessment results need to be 
shared and discussed by key stakeholders to 
develop hypotheses for resulting outcomes and 
recommendations for subsequent actions to 
be taken in the spirit of improved teaching and 
learning. 

Consistent data repository — including direct 
and indirect data. Being able to adequately present 
evidence that expected learning outcomes have 
been achieved, as demonstrated by students, 
requires both direct and indirect evidence. Self- 
reported assessments of learning collected via 
survey research have some value, but targeted 
performance-based portfolios or standardized 
tests can take assessments to a higher level in 
outcome-fulfillment validation. However, indirect 
survey-based evidence, in some aspects, is the low- 
hanging fruit at many institutions as they struggle 
in many disciplines to produce direct performance- 
based evidence. Commercial standardized tests 
are expensive and have content limitations in 
the linkage to specified program goals in many 
disciplines and institutions. Indirect evidence of 
student learning can be handled for the most part 
at an institutional level; however, the delivery of 
direct evidence is dependent on the faculty at the 
discipline level. Developing a deepened culture of 
evidence takes years to establish and is an ongoing 
quest. 

Benchmark processes with similar processes 
at other institutions. Benchmark comparisons that 
result from institutional participation in national 
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(external) surveys are an important part of any 
strategic assessment plan, as they provide context 
for the data. Thus, each institution should examine 
comparatively the content of available surveys and 
determine which would yield data best suited to 
its mission, goals, and objectives. It is important to 
have some outside surveys in the assessment plan 
to furnish external context. 

Create internal formal unit review. An 
institutional strategic plan for assessment should 
include some form of ongoing internal review of 
campus units. Perhaps this could be considered 
as a form of "accreditation" and "continuous 
improvement" — a unit at a time (Banta et al., 2009). 
Unit reviews benefit from the perspective of an 
empirical examination overtime (possibly, the most 
recent five years), including key indicators that help 
inform practice. This process applies to not only 
academic units but educational support units as 
well. 

For example, in the spirit of ongoing 
improvement, the internal unit review process has 
been implemented at BYU for well over 1 0 years 
for both academic (since 1997-98) and educational 
support units (since 1999-2000), with many units 
having gone through the process two times. Both 
processes (academic and educational support) 
employ a similar framework wherein units are 
subject to an internal campus review every seven 
years. Campuswide review committees have been 
established for both the academic group composed 
of faculty and the educational support group 
composed of faculty and staff, all of whom serve a 
three-year term. The review process includes the 
unit's writing of a self-study, which is utilized by the 
internal team as well as the external reviewer(s) in 
the discipline or field during the two-day site visit. 
Following the visit, the reports from the internal 
and external reviewers are shared with the units 
and their corresponding vice president, as review 
findings are utilized in subsequent planning 
regarding the unit. This process has helped 
inculcate a culture of assessment and continuous 
improvement at BYU, which greatly helps facilitate 
data-driven decision-making. Effort has been 
made to have units see themselves as the principal 



consumer of the process — hence, they are doing it 
for their own benefit. 

Continuously review and improve the 
assessment plan. Just as educational processes 
can continually improve, so can corresponding 
assessment processes. Effort should be made to 
improve the use of all of the information provided 
by assessment tools, along with the efficiency of 
data acquisition, analysis, and feedback into the 
learning process. At the same time, academic 
units should be continually evaluating additional 
assessment tools for possible adoption into their 
program. Assessment and continuous improvement 
will always be works in progress. Obviously, each 
department needs to develop a strategy that fits its 
characteristics, objectives, and needs. 

Challenges and Strategies 

The principles and examples of BYU's 
assessment plan presented above reflect the ideal 
assessment plan and situation, implemented 
seamlessly without resistance or obstacles to thwart 
its progress. However, this ideal situation did not 
exist, and many challenges and obstacles were 
encountered. The most commonly encountered 
challenges included the following: 

• Quality of learning outcomes; 

• Alignment of outcomes; 

• Culture of assessment disparity across 
campus; 

• Lack of stakeholder engagement; 

• Lack of collecting direct and indirect 
evidence; and 

• Evidence of using the assessment data. 

Quality of Learning Outcomes 

One of the first challenges that presented 
itself was getting faculty and administrators to 
establish quality expected learner outcomes at 
all levels (institutional, program, course, etc.) in 
order to develop assessment plans that would 
yield results to suitably inform about student 
learning and meet the requirements of external 
and internal accountabilities. These include issues 
such as the need to change from a faculty-centric 
to a student-centric language of stated outcomes. 
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It also includes the difficulty in achieving faculty 
consensus within departments on a tight set of 
learning outcomes (i.e., which courses map to 
primary program goals). 

This challenge was addressed as a lack of 
training or ability among faculty to write good 
learning outcomes at the program or course level. 
Program-level expected learning outcomes needed 
more specificity than "find a job or get accepted to 
graduate school." Some of the difficulty had to do 
with the difference between a goal and an intended 
outcome, while other difficulties related to coming 
up with desired outcomes that are measurable — 
connected to specified direct or indirect evidence. 
There was also the temptation to stick with or over- 
emphasize measurable outcomes at the expense of 
less measurable ones. 

Common mistakes made when writing expected 
learning outcomes included the following: 

• Lack specificity — Outcomes provide no idea of 
what knowledge and skills a graduate would 
have after going through the program (e.g., 
only learning outcome is "Be prepared to 
enter a Ph.D. program"). 

• Written at the wrong level — Learning 
outcomes are not program-based (e.g., "Be 
examples of life-long learning and service"). 
These outcomes are more personal goals 
than program learning outcomes. 

• Topical — The learning outcomes are only 
a listing of the major program topics, not 
what the student will learn about them or 
what skills the student will develop (e.g., 

"[a] Facility Leadership, [b] Human and 
Environmental Factors, [c] Planning and 
Project Management"). 

• Lack measurability — The learning outcomes 
listed are not measurable; they do not answer 
the questions of how much, for how long, 
etc. (e.g., "educate university students about 
principles of marriage, parenting, and human 
development"). 

• Are merely descriptive — The learning 
outcomes only describe the program, instead 
of defining what will be accomplished by 
completing the program (e.g., "The Graphic 



Design BFA program is a four-year curriculum 
designed to prepare students to succeed 
as working professionals in the design 
community"). 

• Are too generic — The learning outcomes 
are too generic to provide any level of 
understanding to students trying to make 
decisions about the program (e.g., "Students 
can apply their knowledge to address 
fundamental questions in their area of study"). 

Addressing Inadequate Learning Outcomes 

Administrators recognized the need to assist 
faculty and administrators in writing learning 
outcomes. Specifically, they sought to assist faculty 
to connect their own assessment plan to those 
outcomes. Thus, university leadership involved the 
Center for Teaching and Learning (CTL) to assist 
faculty in this process. To further assist faculty in 
developing learning outcomes, the CTL developed 
and distributed Guidelines for Writing and Assessing 
Program-Level Expected Learning Outcomes across 
campus to all academic units. Related consultations 
were made available from trained CTL personnel 
(see Middle States Commission on Higher 
Education, 2003; Nichols & Nichols, 2000). 

Leading questions also helped facilitate the 
process of writing learning outcomes. Examples of 
leading questions (that were used at BYU) to assist 
faculty in developing learning outcomes include the 
following: 

1 . What qualities and capabilities do you strive 
to foster in your students? 

2. What is the most important knowledge that 
your students acquire from your field of study 
or from working with you? 

3. How does your field of study or your work 
change the way students view themselves? 

4. In what ways does your field of study or what 
you do contribute to a student's well-being? 

5. How does your field of study or what you 
do contribute to the well being of society at 
large? 

6. How do people in this area of study differ 
from those in other areas (knowledge, skills, 
and/or values)? 
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7. How do we know the extent to which 
students are learning what we hope from our 
field of study? 

8. How do we use information about student 
learning and development to enhance 
student learning? 

Alignment of Outcomes 

Compiling, documenting, and publishing 
the learning outcomes of hundreds of programs 
at a large research institution is a significant 
accomplishment in and of itself. However, the 
learning outcomes at any level (course, program, 
institution) need to do more than just stand by 
themselves. As the course threads are woven 
into the fabric of what a degree represents from 
a given program and institution, a cohesive 
alignment needs to be inherent as course-level 
outcomes to program-level outcomes to university- 
level outcomes. All courses should in some way 
contribute to the substantiation of program-level 
and university-level outcomes. This doesn't just all 
happen naturally, as faculty exercise their academic 
freedom to develop and teach their classes. A 
probable cause of this challenge may be a lack of 
effective communication across academic currents 
as often the right hand is not aware of what the 
left hand is doing. Effective alignment has to 
be done by design with collaboration among all 
academic levels within the university. Continued 
allocation of resources for existing courses with 
little or no alignment to program-level goals may be 
deemed questionable in light of strategic learning 
objectives. It is an ongoing challenge of a high 
order to facilitate the strategic roll-up and roll-down 
of learning outcomes at the course, program, and 
university levels. 

A number of strategies were applied at BYU to 
facilitate the alignment of outcomes at the course, 
program, and institutional levels. These efforts have 
also resulted in an effective electronic publication 
of outcomes and the capacity to assist faculty in 
writing learning outcomes. 

• A component of the university's institutional 
assessment strategy focuses on supporting 
program-level documentation. BYU offers 



degrees in over 400 academic programs, an 
enormous task to ramp up and maintain. 
These programs are at the very heart of what 
matters most to the university — students 
acquiring the knowledge and skills to be 
competent in a profession or academic 
discipline. 

• To ensure the success of these programs, it 
is critical that each program be adequately 
documented. Not coincidentally, thorough 
documentation and publication of that 
documentation are also cornerstone 
requirements of the accreditation process 
in the United States. Regional accrediting 
bodies (for BYU in the Northwest Region, it 
is the Northwest Commission on Colleges 
and Universities [NWCCU]) are increasingly 
insistent that institutions have clearly 
documented expected learning outcomes 
for each of their academic programs. 
Additionally, they require that each program 
have a legitimate assessment and evaluation 
strategy for measuring student achievement 
of outcomes. Finally, they expect that those 
responsible for each program will regularly 
use assessment data to improve academic 
programs. 

• BYU, like most other colleges and universities, 
did not have a process or set of tools in place 
to manage program-level documentation. 
BYU launched a learning outcomes wiki to 
facilitate the convenient management and 
publication of program documentation 
(https://learningoutcomes.byu.edu/). Within 
the first year of the wiki site's activation, all 
programs had been documented. Since its 
launch, the site has received nearly one half 

a million views by approximately 1 00,000 
unique visitors. The nearly 500 authorized 
wiki editors have made about 10,000 
changes to the documentation (this number 
dramatically understates the actual editing 
activity on the wiki because logins persist 
indefinitely, and many editing activities occur 
"offline"). 
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• All students during both the fall and winter 
semesters received a personalized email 
invitation from the President of the University 
inviting them to review the expected 
learning outcomes for their majors and 
provide feedback. More recently, the site has 
been enhanced to include the capacity to 
document course-level objectives and their 
connectivity to associated program-level 
outcomes. 

• This web portal also facilitates a visual course- 
level mapping to program-level expected 
learning outcomes to specific classes. This 
enables students to more clearly understand 
the classes in which they are expected 

to acquire specific skills and knowledge 
pertinent to the degrees they are pursuing. 

Refinement of the content being published 
for each of the degree programs has been the 
result of significant efforts of not only faculty, but 
students as well. The student role in the process 
has been extremely valuable, as students are the 
"consumers" of education and the reason we have 
universities. Students were given the opportunity 
to provide feedback concerning clarity and content 
for all degree programs. This feedback mechanism 
was embedded within the wiki site. Students were 
enthusiastic about the prospects of moving from a 
discussion of what classes they took to the specific 
outcomes resulting from their education as they 
considered employment or graduate school. 

Culture of Assessment Disparity Across 
Campus 

A significant challenge in engaging students in 
assessment was the varying depth and maturation 
of a culture of assessment in colleges and their 
respective programs across campus. An internal 
study of graduating students (BYU Senior Survey 
2006-07 & 2007-08) conducted by the BYU 
Institutional Assessment and Analysis Office 
revealed students in colleges that have been 
responding to specialized accrediting bodies for 
years in fields such as engineering, education, and 
nursing were significantly more likely to be aware of 
the expected learning outcomes in their programs 



than students in the social sciences, humanities, and 
other soft science disciplines (see Figure 1 ). 

Educating the students. The BYU College 
of Humanities addressed this challenge by 
implementing an effective strategy to make 
students more aware of the expected learning 
outcomes within their majors. The college 
embedded within the existing university electronic 
teacher/course rating system a few questions 
specific to expected learning outcomes. This 
enabled students to click on a hotlink to the 
outcomes of their major, on the fly, to review them 
before responding to related questions in light of 
the particular course and teacher they were rating. 
This has enabled the College of Humanities to 
realize reasonable gains in student awareness of 
learning outcomes. 



Percent of exiting seniors indicating YES they were aware of 


the learning outcomes for their major by college 


International & Area Studies 


37% 


Management 


42% 


Physical & Math Sciences 


43% 


c Social Sciences 


43% 


o 

Life Sciences 


47% 


I Fine Arts & Communications 


51% 


e Humanities 


53% 


g 




^ Health & Human Performance 

e 


55% 


Education 

Fnninpprinn & Tpr’'hnrJr»n\/ 


77% 


Li iyii icci M iy Gc i cLi i iiuiuyy 

Nursing 





Figure 1 . BYU student awareness of learning 
outcomes by college. 



Lack of Stakeholder Engagement 

Between the demands of teaching, scholarship, 
and university citizenship, most faculty would 
readily tell how their busy lives have little room for 
much else. They generally like to be left alone with 
their classes and research. As for assessment, many 
feel they already have it covered as they give grades 
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in their classes. It's easy for them to rationalize away 
any assessment responsibility beyond grades to 
their colleagues working in administrative faculty 
roles. They hope that regional and specialized 
accreditation demands won't intrude too much 
on their worlds — that someone else will take 
care of it. They especially are reluctant to expend 
any inordinate amount of energy that doesn't 
seem to contribute to their progression in the 
rank and tenure process or in furthering their 
research agendas, especially without any extra 
compensation. The principal reason we have 
institutions of higher education is for the students 
and their learning, and since faculty are the front 
line crusaders in this effort, it is critical they are 
engaged in the assessment process. Faculty 
members not only need to be aware of how their 
classes contribute to program-level and university- 
level intended outcomes, but need to be invested 
in and accountable to the process. It's really up 
to them to facilitate the collection of appropriate 
evidence of their students' learning. 

Engaging the stakeholders. A number of 
strategies emerged that proved successful in 
engaging stakeholders in the assessment plan. 

• For the most part, a strategy was 
implemented wherein existing campus 
committees, councils, and infrastructure 
(Dean's Council, Associate Academic Vice 
President Council, President's Council, 
Curriculum Council, etc.) were utilized to 
make assessment part of the status quo 
faculty and employee job — in short, the 
normal flow of things and not something 
new or additional, to the degree possible. 

The process was kicked off with high-level 
support from the university President at a 
regularly scheduled annual leadership retreat. 
This included some unrehearsed dialogue 
around learning outcomes awareness and 
fulfillment with students and the deans of 
the colleges of their majors. Academic vice 
presidents, associate deans, department 
chairs, faculty, administrators, and students 
comprised the university task force that was 
established. 



• Rich discussions within existing academic 
faculty councils resulted in many faculty- 
driven initiatives. For example, the university 
curriculum council enhanced the process 
for adding a new program at the university 
to require the inclusion of stated "Expected 
Learning Outcomes." 

• Additionally, faculty councils initiated and 
implemented a cross-college peer review of 
the quality of expected learning outcomes 
statements. Many departments held focus 
groups with their students to refine their 
learning outcomes statements with an eye to 
making them understandable and useful to 
students. 

• Students on the institutional expected 
learning outcomes task force were 
commissioned to create a video (available 
at www.learningoutcomes.byu.edu) to help 
promote awareness and understanding 

of learning outcomes among students 
throughout the university. They created an 
effective product in just a few months. The 
video is available for all units across campus 
as a tool to inform students of the benefits 
of understanding what their respective 
programs expect them to learn and be able 
to do when they graduate. The video also 
illustrates the utility for open (or undeclared) 
major students in selecting a major program 
that aligns their interests with the expected 
learning outcomes. The video vividly profiles 
actual students with the faculty in their 
majors, with commentary from the college 
deans. This video was recently shared at a 
national assessment in higher education 
summit. 

• As all departments met the deadline to 
populate the learning outcomes wiki 
site, all students received a personalized 
email from the university President to visit 
learningoutcomes.byu.edu. 

• Expected learning outcomes across campus 
were not only published electronically on 
the learningoutcomes.byu.edu wiki site, but 
also were posted as hard copies in campus 
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areas highly frequented by students, such as 
advisement centers, and program, department 
and college offices. Publishing the expected 
learning outcomes on campus, especially in 
hard copy form, resulted in added attention 
from faculty. This not only resulted in faculty 
coming to agreement as to the key expected 
learning outcomes for each program, but also 
improved the quality of the statements. It 
seems that there is no better form of faculty 
motivation than public scrutiny. 

• Students were involved in this process as 
well; in addition, the student advisory council 
took the lead in distributing the hard copies 
and in explaining their potential utility to 
students across campus. Students reported 
that many were excited to have a clearer 
high-level understanding of the target goals 
for the many degrees offered across campus. 

Lack of Collecting Direct and Indirect Evidence 

It is a significant challenge to provide evidence 
of student learning especially across the breadth 
and depth of numerous degree programs at an 
institution. There are many ways to assess learning 
outcomes for students. The characteristics of 
good evidence include (a) both direct and indirect 
methods for gathering data and (b) the appropriate 
use of quantitative and qualitative data. Many rely 
mostly on indirect evidence, which can be more 
readily obtained; however, good strategic plans 
for assessment include the systematic collection 
and utilization of direct evidence. It is also good 
to understand the fundamental concepts of 
formative assessment, summative assessment, and 
benchmarking. Convergence from multiple sources 
and types of data is important in substantiating 
whether or not expected learning outcomes have 
been realized (Middle States Commission on Higher 
Education, 2003). 

The concepts of direct and indirect methods of 
evaluating student learning are often confused with 
each other and with quantitative and qualitative 
forms of information. Each has its merits and 
drawbacks. 



Direct evidence demonstrates actual learning, 
performance-based work that is reflected in an 
evaluation of something the student has actually 
done or produced (e.g., homework, tests, papers, 
standardized tests, performances, products). 

Indirect evidence is non-performance-based data 
that reveal characteristics associated with learning, 
but only imply that learning has occurred (e.g., 
self-report survey data, senior survey data, alumni 
questionnaire data, job placement rates, satisfaction 
rates). 

Examples of direct and indirect measures of 
student learning at the program level are presented 
in Figure 2. 



Direct (Primary) Measures 


Indirect (Secondary) Measures 


• Capstone projects, senior 
theses, exhibits, or 
performances 

• Pass rates or scores on 
licensure, certification, or 
subject area tests 
(Educational Testing Services) 

• Student publications or 
conference presentations 

• Employer and internship 
supervisor rating or students' 
performance 

• GRE scores of students 
completing the program 

• Performance-based skills 
competency exams 

• Portfolios mapped to 
intended outcomes with 
embedded rubric 
(criterion-based rating scale) 
ratings 


• BYU Alumni questionnaire 
data 

• BYU Senior survey data 

• National Survey of Student 
Engagement (NSSE) data 

• Employer surveys 
(non-supervisor) 

• Graduate school placement 
rates 

• Departmental exit interviews 

• Portfolios with no 
performance-based linkage 
to intended outcomes 

• Focus group interviews with 
students, faculty, employers, 
etc. 



Figure 2. Direct and indirect measures used at BYU. 



Institutional strategies should be developed 
and put into action that results in the appropriate 
collection of direct and indirect evidence. A 
number of questions should be considered: How 
will each learning outcome be assessed? What 
evidence of student learning is most relevant for 
each learning outcome? What criteria will be used 
to evaluate this evidence? What constitutes student 
performance at exceptional, acceptable, and 
unacceptable levels? 
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An effective strategy should include the 
systematic and routine collection, storage, analyses, 
and utilization of evidence pertaining to each stated 
learning outcome. Such data could be collected via 
embedded classroom assessments or institutionally 
driven assessment initiatives. It should use multiple 
kinds of assessments for each learning outcome, 
including direct outcomes, such as outcome-related 
course assignments, exams, and capstone course 
projects, as well as indirect outcomes, such as 
employer, alumni and senior surveys (Association 
of American Colleges and Universities, 2004; 

Middle States Commission on Higher Education, 
2003; Miller & Leskes, 2005). For example, new 
functionality in the university's online student 
ratings system allows colleges and departments to 
add custom questions to the standard evaluation 
form. Additionally, questions about learning 
outcomes have been integrated into the university 
senior exit survey administered annually to all 
students receiving undergraduate degrees. 

Resulting data are shared annually at all academic 
levels. Academic departments and programs also 
have latitude to add custom questions to annual 
alumni and senior exit surveys that, if desired, 
could connect specifically to the department's or 
program's expected learning outcomes. 

Evidence of Using the Assessment Data 

Using assessment results is important as units 
move to close the loop and experience (hopefully) 
a positive return on their investment as they 
seek to improve. Notwithstanding, this is and 
continues to be the biggest and most significant 
challenge within the assessment arena at virtually 
all institutions of higher education. It speaks to the 
return on investment in assessment, and university 
stakeholders rotate around the assessment loop 
in response to the questions of "What difference 
does all this assessment make?" and "Is it worth 
it?" It behooves all institutions to have an effective 
connection between their assessment and planning 
processes (Middaugh, 2009). Ultimately, better 
planning decisions are made when corresponding 
assessment data are utilized. Although most 
regional accrediting bodies, to some degree, ask 



universities to provide evidence that assessment is 
making a difference and, in fact, assessment loops 
are being closed, many institutions struggle to 
deliver such evidence. All such evidence should 
point to increased learning among students. 

Institutions need to determine what makes 
most sense in their culture and environment in 
establishing a framework for documenting the use 
of assessment data. Setting an appropriate campus 
tone as to expectations and frameworks for such 
accountability is critical if this is to be accomplished. 
A number of strategies could be applied to provide 
a means to systematically collect and utilize such 
data, and these are outlined below. 

• Have high-level university leadership endorse 
and promote the expectation for units to 
document their utilization of assessment data. 

• Embed an accountability framework with the 
self-study guidelines for internal unit review 
with the university. 

• Create documentation channels with the 
resource planning or allocation processes 
within the university. 

• Develop other data collection processes to 
systematically obtain evidence of assessment 
data usage across campus. 

• Take advantage of any existing accountability 
channels on campus that would facilitate 
documenting closing the loop such as annual 
faculty stewardship interviews. 

BYU has made strides in this area, but ongoing 
improvement in documenting "closing the loop" 
is continually sought. This applies to not only 
academic units but educational support units as 
well. One college at BYU utilized the results of 
the university annually administered survey to 
alumni (three years post-degree) to enhance the 
learning for their students. Resulting data showed 
their alumni to be lower than most BYU graduates 
in the constructs of "quantitative reasoning" and 
"uses technology effectively." Upon sharing some of 
these data with their National Advisory Council and 
further discussions, they decided to initiate efforts 
in their college to enhance student learning and 
development in these areas. Another educational 
support unit at BYU, the Student Health Center, was 
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able to greatly enhance the health benefits for all 
students based on findings from a student survey. 
Upon learning that students not only wanted but 
were willing to pay for additional dental and vision 
benefits, potential options were explored, and this 
exploration ultimately led to benefits being added 
without any additional cost to students — triggered 
by the survey's indirect evidence. 

Collegewide Application of an 
Assessment Plan 

An example of using the principles of strategic 
planning discussed above is found in The David O. 
McKay School of Education (McKay School), whose 
teacher education programs involve 8 colleges 
and 24 departments on the BYU campus, and 
comprises nearly 6,000 students of the institution's 
total enrollment that surpasses 30,000. The 
assessment system discussed in this section applies 
to the following departments: Special Education, 
Elementary Education, Secondary Education, and 
Early Childhood Education. This section will focus 
on an assessment plan from a collegewide and 
individual program perspective; more specifically, 
it will discuss purposeful assessments, analysis, and 
reporting as related to expected learning outcome 
of the School of Education and its departments. 

The primary purpose of any assessment plan 
among academic disciplines is to establish evidence 
of learning outcomes or program validity. To 
achieve this end in the McKay School, the following 
six critical and integrated principles underlie this 
college's system approach to assessment, analysis 
and reporting: 

1 . Align claims or the conceptual framework 
of the academic program with assessment 
measures. 

2. Establish a master calendar that is agreed 
upon by all stakeholders. The master calendar 
should clearly state when assessments 

are administered, results are analyzed and 
reported, and responses to reports are due 
during the academic year. 

3. Establish a database and website to provide 
accessible assessment data and reports for 
stakeholders. The website provides access to 



the database, which contains all of the core 
assessment data from the teacher education 
programs. The website also provides access to 
completed reports of assessment data. 

4. Tell the data story in a timely, consistent way. 
The data analysis and reporting follow each 
reporting cycle and provide feedback to 
the faculty and program leaders, especially 
on considerations to make program 
improvements. 

5. Create a culture of evidence and learning 
partnerships by engaging stakeholders not 
only as owners in the establishment who use 
the master calendar, but more importantly 
as owners of the assessment, analysis, and 
reporting system. 

6. Include a way for faculty and program leaders 
to respond to or demonstrate how the data 
are being used for program changes and 
improvements. This not only affirms a culture 
of evidence, but also can lead to program 
validity and improvements. In other words, 

if we don't tell the data story and encourage 
accountability to use and interpret the 
results, how do we know that program aims 
and student learner outcomes have been 
achieved? 

Following is a more in-depth discussion of these 
critical factors of an assessment plan as used in the 
McKay School. 

Alignment of Assessment Instruments to 
Programs' Conceptual Framework and 
Outcomes 

The teacher education programs of the McKay 
School spent a considerable amount of time 
developing their conceptual framework. This 
conceptual framework unifies the institution's 
aims and mission, the aims and mission of the 
School of Education, and important aspects of 
the teacher education program as found in the 
learning outcomes (http://education.byu.edu/epp/ 
framework.html). Assessment measures used in the 
teacher education program have been designed 
to align the program's conceptual framework. 

These instruments are used at a variety of points 
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throughout the students' training in the program 
such as, to determine student admissibility 
(admissions) to the teacher education program, 
to assess achievement of learner outcomes and 
student readiness during the pre-clinical practice 
(practicum), and to record effectiveness of 
performance during the student teaching (clinical) 
experience. Surveys have also been developed to 
ask seniors and alumni to evaluate their teacher 
education program. As well, employers are also 
asked to evaluate McKay School teacher education 
graduates regarding their preparation and 
performance as full-time teachers. 

The primary core assessment measures in 
the teacher education programs are designed to 
measure different traits, dispositions, and skills. 

Two of the measures are designed to assess 
dispositions, and both are administered during the 
admissions process and the clinical experience. 
Another measure is designed to assess performance 
during the pre-clinical and clinical student teaching 
experience; it is based on the Interstate New 
Teacher Assessment and Support Consortium 
(INTASC) standards. The INTASC Standards are 



model standards of values and traits of good 
teaching and have been developed by the 
Council of Chief State School Officers (CCSSO) 
to help create consensus on good teaching 
practices (http://www.ccsso.org/Projects/ 
interstate_new_teacher_assessment_and_ 
support_consortium/780.cfm#resource).The 
INTASC Standards play an important part in the 
Teacher Education's conceptual framework. The final 
assessment measure is students' capstone project 
and is completed during the clinical experience; it 
is graded using a specific rubric designed for that 
purpose. 

The Assessment, Analysis, and Reporting 
Master Calendar 

The master calendar is the core of the School of 
Education assessment plan. In the calendar, there is 
an emphasis by date on the (a) chronological set-up 
and identification of the instruments to be used; 

(b) dates on which the scores are to be recorded; (c) 
dates when the recording of data is to take place; (d) 
analysis and reporting period; and (e) department 
or program response to data (see Figure 3). 



ASSESSMENTS MASTER CALENDAR 
Department 



INTENTIONAL 

INSTRUMENT 


SET-UP 


RECORDING 


REPORTING 


PROGRAM 

RESPONSE 


Instrument 


Date 


Date 


Date 


Date 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






" 


" 


" 






“ 


" 


" 






" 


" 


" 






" 


“ 


" 




« 


" 


" 


" 




a 


" 


" 


" 




n 


" 


" 


" 




a 


" 


" 


" 



SEMESTER REPORTS Date 

ANNUAL INTEGRATED AND COMPREHENSIVE REPORT Date 
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Figure 3. BYU assessments master calendar. 
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The calendar was developed through consulting 
the governing committee of the teacher education 
programs. This helped create a sense of ownership 
among the committee and, thereby, department 
faculty. Naturally, the implementation of this 
calendar has not been seamless, and creating 
buy-in among stakeholders has been a difficult 
process and one that requires constant vigilance. 
However, prior to the use of the calendar, data 
would gradually trickle in each semester, or would 
not be turned in at all. There are still the perennial 
stragglers in submitting their data each semester, 
but the process has improved markedly and has 
enabled more accurate and timely reporting of 
the assessment data. The calendar represents 
agreement on timelines among program faculty to 
keep the assessment plan moving forward. 

Accessible Data via a Website and Database 

Previous to the development of a database 
and website, the teacher education program's 
assessment data was stored in multiple files and 
locations. Accessing data, let alone analyzing the 



data, was nearly impossible for stakeholders. To 
address this issue of inaccessible data, a database 
was developed to house the information in an 
organized, usable format. In addition, a website 
was created to allow online access to the database 
as well as to all reports and the master calendar. 

The idea was to have the data, reports, and master 
calendar accessible in the same location to reduce 
the number of places stakeholders had to go 
to get the information they needed. Through 
prior experience of trying to disseminate raw 
data and reports, it was found that if access to 
this information required stakeholders to go to 
multiple locations or required several steps to get 
them there, they were much less likely to use the 
information because of the extra time and effort 
just to find it. As a result, the raw data was placed in 
a database and the reports were put online through 
an assessment website. The graphic in Figure 4 
displays the website access to both the reports and 
database from one location, and Figure 5 shows 
the report database and the database that stores 
the raw data. In the report database, stakeholders 



David O. McKay School of Education 

Center for the Improvement of Teacher Education and Schooling 

Home Reports Database TEAC Materials Master Calendar Assessment, Analysis, Reporting, and Research Team 










Welcome to the Educator Preparation Program 
Assessment, Analysis, Reporting, and Research 
website. On this website you will be able to do 
the following: 

O Reports: 2007-2008 academic year forward 

O Database: Stores data from Fall 2004 
forward 

O TEAC accreditation materials 

O Master Calendar 

O Principles, Purpose, Processes, and 
Products of the Assessment, Analysis, 
Reporting, and Reseach Teams 

The Assessment, Analysis, Reporting, and 
Research Team can be reached by clicking here. 



Figure 4. The McKay School's assessment website and database. 
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Report (hit ii base on the website 



The following reports were found for your particular query; 

* 4 Annual Reports jbr Academic years 20XX. 20XX. 20XX and 20 XX 

* 2 Semesters Reports for Winter 20XX and Fall 20XX 

* 3 Retention Reports for academic years 20XX. 20XX and 20XX 

* 1 Graduation Report for Academic year 20XX 



Du tn base accessed through website 



Query Students by 



Search Da fa ha sc by 



Any Academic Year 
Any Semester 0 
Any Program 



0 



0 



Show Reports 



Student ID 





Last 


a 


First 


a 



Year 


Semester 


Course 


I Instructor 
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Figure 5. The McKay School's assessment report database. 



have the option of searching by year, semester, 
and program. In the other database (that contains 
raw data), stakeholders have many more criteria 
available for searches, as shown in the lower section 
of Figure 5. 

The database where raw data are stored 
also enables stakeholders to query and export 
the data for their specific needs. Access to data 
and reports means that stakeholders can readily 
obtain information regarding program strengths 
and weaknesses, along with general trends in 
assessment scores. Both the database and website 
have been very well received by stakeholders. 

Whether access to the data is enabled through 
a database, website, or other data management 
system, making data readily available to 
stakeholders (or telling the data story) is a key 
component to an assessment plan. 

Data Analysis and Reporting 

Reporting tells the data story, and assessment 
results must be communicated usefully, clearly, 



and accurately. Large, cumbersome reports filled 
with too much data, even if it is good data, are 
rarely read or used. In order to avoid creating 
any additional reports that might fall into this 
classification, assessment administrators sat down 
with faculty and program leadership to obtain their 
feedback regarding what kinds or reports would be 
most helpful to them to make informed decisions 
about program improvements. Through engaging 
stakeholders, assessment administrators were able 
to develop analysis templates for the semester 
and annual reporting. In addition to developing 
templates that satisfy needs for reporting program 
progress, they were also able to incorporate 
data to meet accreditation-reporting needs, 
thereby fulfilling two needs in one report. Figure 
6 is an example of one of the semester-reporting 
templates that was developed as a result of 
engaging stakeholders about their reporting needs. 
As mentioned in the previous section, stakeholders 
are able to access these reports online for their 
specific department and/or college. 
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Instrument - Final Report 



Department 

Date 



Instructor Evaluator 



Course Enrolled Recorded %Recorded Max Min 

423 420 99% 5.00 1.00 



Evaluation 
Item 1 
item 2 
Item 3 
Item 4 
item 5 
Item 6 
Item 7 
Item 8 
Item 9 

Item 10 



M 

4.45 
4 31 
392 
4,33 
4.23 
429 
4.42 
4.09 
4.67 
4.55 



SD 

061 

069 

0.74 

0.67 

0.77 

0.69 

0.66 

0.70 

0.55 

0.63 



Overall 



4.33 



0.70 



Figure 6. Report template example. 



Lessons Learned and Future Strategies 

Certainly, there is no universal remedy in 
meeting the challenges of establishing an 
assessment plan. Our intent, however, in this article 
was to share a working model of a systematic 
approach to creating and establishing a meaningful 
assessment culture, as well a successful assessment 
plan (i.e., doing assessments that really matter 
and make a difference). While there will always be 
institutional constraints, the campus commitment 
and will to see students as first in setting and 
meeting educational aims should be unrestrained. 
The need to know students' needs, expectations, 
and success with learning outcomes and the 
importance of caring about their achieving success 
in the teacher education program are more cultural 
than resources-related. 

Without ongoing learner outcomes assessment 
or continuous monitoring of student progress, 
the likelihood of both knowing and obtaining 
student — and, thereby, program — success is fragile. 



Yet student success, the optimal experience on 
the campus, is everyone's business. Via a systems 
approach to assessment, analysis, and reporting, 
student success is something we make happen. 

Essentially, the primary lessons learned from the 
institutional assessment and analysis plan presented 
in this paper are threefold: 

• While our current campuswide assessments 
and analysis is a work in progress, it requires 
both patience and constant vigilance to 
obtain consistency in assessments and 

to communicate their meaningfulness to 
stakeholders; 

• Celebration of successes, when obtained, 
is important; yet, at the same time, it is 
essential to recognize where adjustments in 
the assessment processes need to be made. 
Refinements to the system, we have learned, 
ought to be done by listening to stakeholders 
and then thoughtfully outlining next steps to 
improve the assessment plan, 
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• And perhaps most importantly, we have 
learned that effective assessment really 
touches and is dependent in some way upon, 
everyone on campus — the students, faculty, 
and staff. A culture of assessment doesn't 
happen overnight at the flick of a switch. It 
doesn't usually just evolve by itself either. It 
takes planning, leadership, resources, and 
effort. The experiences that individuals have 
had personally or vicariously within their 
campus environment ultimately shape their 
beliefs about varying issues or movements 
and subsequently impact their actions and 
the potential results of any initiative. Thus, 

an effective means to transition a campus 
to an enhanced culture of evidence is to 
provide new opportunities that allow campus 
constituents to have meaningful experiences 
with data, assessment, and the like, and to 
genuinely feel they made a positive difference. 

At the end of the day, we've also learned that 
assessment is a mechanism for promoting and 
managing ongoing change and improvement in 
the organization. It is an essential function of the 
higher education, and specifically the BYU, learning 
enterprise. While it has not been an easy path or a 
yellow brick road of implementation, we've come 
to appreciate and value (a) assessments that really 
matter and make a difference in student learning; 
(b) purposeful or intentional assessments that 
respond effectively to a variety of external and 
internal forces; and (c) assessments — using data 
to improve programs and student learning and 
success — as a mechanism of change that must 
directly involve stakeholders. These paths we will 
continue to pursue. 

Other factors, we've learned and attempted to 
incorporate, of an assessment culture are involved 
with the following: 

• Creating learning partnerships in the 
assessment of student development, growth, 
and success factors; 

• Aligning conceptual frameworks with the 
claims of higher education programs and the 
assessments/evidence that support them; 

• Engaging faculty and students in the 
assessment process; 



• Using technology (including electronic 
portfolios) to assist the monitoring and 
tracking of student development and success; 

• Accounting for and addressing matters of 
diversity, including diverse learners; 

• Using research lenses to assemble the 
assessment puzzle — who students 
are (internal and external conditions 
and characteristics), what students do 
(engagement), and what institutions do 
(policies, procedures, structures, processes); 

• Developing assessment models that lead to 
program improvement and student growth; 
and 

• Bridging assessment, learning outcomes 
and accreditation — a systems approach (see 
Angelo, 2002, 2007; Astin, 1992; Ewell, 2007; 
Hollowell, Middaugh, & Sibolski, 2008; Pet- 
Armacost & Armacost, 2008; Suskie, 2007; 
Volkwein, 2008) 

Essentially, what do good programs do to 
achieve an assessment culture? What really 
matters in higher education assessments? 

What are the common assessment ingredients 
found among successful programs? What does 
the research suggest as next steps that higher 
education programs should consider in promoting 
qualitative assessments of the teaching-learning 
process? McDonald (2002) attributes the concept 
of a campus community to Ernest Boyer, a highly 
respected and influential higher education scholar. 
Over his career, Boyer, former President of The 
Carnegie Foundation for the Advancement of 
Teaching, consistently advocated that all parts of 
campus life — from the recruitment of students 
to teaching — contribute to students' sense of 
"wholeness." He advocated that the "true calling" 
for higher education is to create a community that 
includes everyone on the campus and to foster the 
success and dignity of each person. In this context, 
then, what matters in assessment and to student 
success is a student-centered assessment culture. 
Bailey (2006) suggests the following: 

• Most students benefit from early 
interventions and sustained attention at 
various transition points in their educational 
journey. 
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• Students who make worthwhile connections 
with something or someone in the 
postsecondary environment are more likely 
to engage in educationally purposeful 
activities during college, to persist, and to 
achieve their educational objectives. 

• Institutions that focus on student success and 
create a student-centered culture are better 
positioned to help their students attain and 
achieve their educational objectives. 

• Because we value what we measure, 
institutions should focus assessment and 
accountability efforts on what matters to 
student success — institutional effectiveness 
and student success will not improve without 
valid, reliable information to guide change 
efforts and monitor performance. 

If higher education leaders are serious about 
promoting an assessment culture that is sensitive 
to student and program needs and dedicated to 
improvement, they will strive to align student and 
program expectations, enrich students'development 
through viable learning opportunities, and monitor 
student and program achievement of goals through 
assessment of learning outcomes. 

In conclusion, it was our intent to demonstrate 
that intentional assessments in higher education 
ought to guide strategic planning, analyze cost 
effectiveness, assist in accreditation, measure 
student learning outcomes, and guide decisions, 
policies, and practices. As pointed out by Schuh, 
2007, Kuh, Kinzie, Schuh, and Whitt (2005), and 
again by Kinzie and Kuh (2007), many institutions 
have created a culture in which data-driven 
decisions lead to continuous improvement. These 
institutions, they point out, rely on systematic 
information to make good decisions. Moreover, 
in their research project Documenting Effective 
Educational Practices, the institutions studied 
routinely collected data and made decisions about 
beginning, ending, or modifying programs based 
on evidence. Many institutions that have developed 
an assessment culture have done so, in part, due 
to accrediting agencies' insistence that higher 
education programs use data to demonstrate that 
students are learning and are achieving the claims 



of the program. Specifically, assessment is essential 
for both external and internal reasons. Assessment 
is especially effective when it measures student 
learning, success, and program effectiveness. Just as 
important, however, is the use of assessment data 
to drive program decisions that improve student 
learning and success. 

A premise of this paper is that all components 
of the higher education community, especially the 
college and department community, are part of the 
student learning enterprise; that each is dedicated 
to student learning, growth, success; and that this is 
the business of everyone in the campus community. 
Specifically, effective assessment is about building a 
culture that anticipates the needs of students who 
are entering the system and is concerned with their 
progress in it. 



Editor's Note: 

Building a knowledge base would seem to 
involve three key aspects: a motivated community 
of learners with a memory function, a set of 
conceptual models that neither have to either 
agree with each other nor have to cover the entire 
domain of knowing, and experiences or data that 
pertain to the knowing. Assessment of academic 
programs has become one of higher education's 
more prominent knowledge bases. Knowing 
about knowing has multiple groups of motivated 
communities. First are those directly involved in the 
academic program, the students and the instructors. 
Next, there are those who manage the cluster of 
programs in a shared process of administration, 
management, and governance with overlapping 
membership of individuals in the various roles such 
as Deans, Department Chairs, Promotion and Tenure 
Committees, and core administrative processes. 
Finally, there are the external regulators from 
accreditation to legislative to the public. 

The conceptual models include the technical 
discussions on how people learn, the role of active 
engaged learning, best practices of teaching, 
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discipline paradigms, and management insights 
from books such as Competing on Analytics, Good to 
Great, and The Fifth Discipline. 

The data, however, most frequently come from 
specific case studies, which continually balance 
internal validity for interpretation and external 
validity for generalization. 

In this article, Kramer, Hanson, and Olsen 
present us with all of the key components of a 
learning community and, in doing so, they produce 
a case study that can serve as data to sustain 
continued learning about assessment. Your role, as 
a member of the learning community, is to consider 
how much it explains the key aspects in your 
institution and what parts of it generalize to your 
situation. 

The discussion looks at assessment primarily 
through the lenses of faculty in their governance 
role and the university in its administrative support 
role. Of course, it also refers to the external force of 
accreditation. How did BYU balance and leverage 
these groups that sometimes have competing 
goals? Would those strategies work at your 
institution? 

The conceptual models often relied on 
grounded theory and had a variance across 
programs. There were some management decisions 
that represent best practices in implementing 
change. There was extensive involvement of faculty. 
There was flexibility to adapt to unique needs. 
These decisions also represented best practices in 
learning. Feedback was built into the system as an 



integral part. Data were made available, and uses 
were documented. What else is consistent with 
what we know about change? For example, how 
involved were BYU senior administrators, and how 
does this compare to your institution? 

The factors above are some of the aspects that 
make this study valuable as a description of how 
to implement change. There are also a couple 
of aspects that make it a very valuable part of a 
roadmap to sustainable assessment. The first aspect 
is that it discusses some of the challenges and what 
was done to deal with them. Note that they don't 
claim to have solved all of the challenges as several 
times the phrase "constant vigilance" is used. Some 
of these challenges are inherent in the academy, 
and some of them are a function of BYU's context. 
Which of these challenges exist at your institution, 
and what are the differences based on context and 
characteristics? 

The second aspect is the discussion of "lessons 
learned and future strategies." Here is a template 
but does it apply to your situation? For example, 
do you celebrate successes? Finally, while these 
again remind us of "best practices" with concepts of 
patience and persistence, integrated involvement 
of community members, and the use of technology 
to enable and facilitate the process, they remind us 
that "quality assessment is a journey."Thanks to this 
description by our colleagues of their journey at 
BYU, the path will most likely be clearer for others of 
us on the same journey. 
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